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0. Introduction 

Let k he a field and n be a natural number. By Gn{k) = XJT^n^k) we denote the group of 
all unipotent lower-triangular n x n-matrices with coefficients from k; this group is called a 
unitriangular group. By 0„(A;) = ut(n, /c) we denote its Lie algebra over k; this Lie algebra 
consists of all nilpotent lower-triangular matrices with coefficients in k. 

If = is a finite field, then Gn{q) = Gn{k) is a finite group; so, there are finitely many 
classes of equivalency of irreducible complex representations of this group. A description of all 
irreducible characters (or of some series of them) is a classical problem of representation theory. 
The orbit method of A. A. Kirillov |K1| . |K2| allows to reduce the similar problem of description 
of unitary irreducible representations of Lie groups to the problem of classification of coadjoint 
orbits; this method is also valid for G'„(g) (see. [KaJ), but complete classification of coadjoint 
orbits for an arbitrary n is unknown. 

A description of orbits of the principal series (i.e., orbits of maximum dimension) of Lie 
groups was presented in the pioneering work on the orbit method |K2j : it's also valid over a 
finite field |K3| . In C. Andre's works (see |A1| . |A2| ) so-called basic characters are described 
(in particular, an exact formula for characters of the principal series is found). 

The problem of description of orbits, representations and characters of sub-maximum di- 
mension is a natural generalisation of these results; such orbits, representations and characters 
are called subregular. They play an important role in algebraic geometry and i^'-theory (see, for 
example, |Lu|). Subregular orbits are described in [IP]. The main goal of this paper is to give an 
exact formula for the corresponding characters. This formula shows that subregular characters 
(as characters of the principal series) can be described in terms of coefficients of minors of the 
characteristic matrix. 

The paper is organized as follows. In section[T]we collect some basic facts about the group On 
and discuss Andre's formulae for characters of the principal series (Theorem ll.l2p . In section [2] 
we formulate the Main Theorem including exact formulas for subregular characters (Theorem 
2.9p . More precisely, for an arbitrary subregular character we find the elements such that the 
value of this character on the conjugacy classes of these elements is non-zero and compute 
this value. Section [3] is devoted to the description of these conjugacy classes (Theorem 13.51) . In 
section [4] we recall some facts about semi-direct decomposition of the group Gn, wich are needed 
for the proof of the main Theorem, which is given in section [5l Finally, section [6] includes some 
remarks about discussed problems. 



*This research was supported by Samara regional grant for students and young scientists JV223E2.1D 
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1. Characters of the principal series 

Let k = ¥q and char /c = p, i.e., q = p"^ , where p is a prime number. Throughout the paper, 
we suppose p^ n. According to |Ka| . under these assumptions the orbit method is vaUd: there 
is a one-to one correspondence between the set of all irreducible complex characters of our 
group and the set of all coadjoint orbits, i.e, G„(g)-orbits in the dual space Qn{q)- Moreover, to 
each orbit VL C 0n(9) the character 

Xf.(expa)=g-^^-^-5^^^(/(a)), aG0„(g), (1) 

/en 

is assigned (here 6': Fg — C is a non-trivial character of the additive group of the field Fg). On 
the other hand, there are no exact formulas that allow to find the value of the character of a 
given orbit on a given element of the unitriangular group. 

Remark 1.1. There exists a form {A^B) = tr^AB) which is non-degenerate on gln{k). 
This allows to identify ut* {n,q) with the space of all nilpotent upper-triangular matrices by 
the formula /(x) = {f,x) = J^ijijiXij, where x = (xij) G Qn{q), f = fej) e Q^iq)- Then the 
coadjoint action 

K{g):Qn^Qn- {K{g)f){x) = fiAdg^ix), g e Gn, feQ^xeQn 

is given by 

K{g) : X ^ {gxg-^)high, x E gl, g e 0„, 

where (a) high denotes the following matrix: its elements below the main diagonal coincide with 
the corresponding elements of the matrix a and its elements on the main diagonal and below 
are equal to zero. 

Representations of maximum dimension play an important role in this theory. These re- 
presentations (and their orbits and characters) form the so-called principal series; orbits of 
maximum dimension are called regular. We'll use the following 

Notation 1.2. Let neN. Set uq = [n/2], rii = [{n - l)/2]. Note that n = no + ni + 1. 

Notation 1.3. Let g = {yij) G Mat{n,k). By Al^'"''l'' (g) we denote the minor of the 
matrix g with the rows Zi, . . . , and the columns Ji, . . . ,jk in the given order (for a given 
1 ^ k ^ n,l ^ ii, . . . ,ik, ji, ■ ■ ■ , jk ^ n). In particular, for an arbitrary 1 ^ d ^ Uq we set 

We also denote A'^{g) = A^|-^^''"'^^j((yf), where X is the set of the pairs X = {(ii, ji), . . . , {ik,jk)} 
and a, r are permutations such that cr(zi) < . . . < a{ik) and r(ji) < . . . < r(jfc). 
A complete description of orbits of the principal series is well-known |K3| : 
Theorem 1.4. Let k = ¥g he a finite field. An arbitrary regular orbit has the following 
defining equations: 

Arf(*a) =(3d, ae gUq), 1 ^ d ^ uq 

(here *a is the transpose of the matrix a, P^i are arbitrary scalars from F^, Pi, . . . , Pno~i ^ IF* 
and Pno 7^ for odd n). □ 

Corollary 1.5. For an arbitrary orbits of the principal series there exists the unique 
canonical form of a regular orbit, i.e., the matrix of the form 

/O ... 

... 6,n-l 
J . . . . ) 

^0 ... y 
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where „ = ^d,n-d+i = -p. — , (i = 2, . . . , rio, that consists in this orbit. □ 

Pd-i 

Moreover, dimension of a representation (and an orbit) of the principal series is known |Le] : 

Theorem 1.6. Let /i(n) = (n — 2) + (n — 4) + . . . and be the representation that 
corresponds to an orbit Vt of the principal series. Then dimf2 = 2ij,{n), dimTn = qf^^"-\ □ 

Note that there are exactly q'^f^^"'^ points on an orbit f2 of dimension 2/i(ri) |K4| . 

In this section we recall the formula for characters of the principal series, i.e., characters of 
the form X = Xfi where / G fi/ C 0* is the canonical form on a regular orbit. More precisely, 
let X be an irreducible character of the group G„. By SuppX denote its support (i.e., the set 
{g e Gn I x{g) 7^ 0})) obviously, the support is a union of certain conjugacy classes and the 
value of the character on an arbitrary conjugacy class is constant. So, it's enough to describe 
the support explicitly and compute the value of the character on an arbitrary conjugacy class 
containing in the support. 

We'll use the following notations from Andre's paper |A2| : 

Notation 1.7. We denote by $(n) the set of all pairs \ l^j<i^n} (we call them 

roots, because to each (i,j) G $(n) the root vector e^j, i.e., the matrix with 1 in the (j, i)-th 
entry and zeroes elsewhere, is assigned). Let D C $(n) be a subset that contains at most one 
element from each row and at most one element from each column; then this subset is called 
basic. If a basic subset consists from the roots of the form {n — j + 1, j) then it's called regular. 

Example 1.8. Here we draw one of regular subsets: D = {(6, 1), ((4, 3))} C $(6). The 
entries (i,j) G D are marked by the symbol CS>: 



1 2 3 4 5 6 













































































Notation 1.9. For an arbitrary basis subset D C $(n) and arbitrary map ip: D ^ ¥* 
consider the element of the group Gn{q) of the form 

Xoi^) = ln+ ^ (p{n-J + l,j)en-j+l,j, 
{n~j+l,j)€D 

where 1„ is the identity n x ri-matrix (ii D = then x^i^p) = In)- By %£){ip) denote the 
conjugacy class of this element and by %reg denote the (disjoint) union of %£){{p) such that D 
is a regular subset and ip: D ^ ¥* is a map. 

Definition 1.10. Let D be a subset of $(n). A root G $(n) is called D-regular, if 

{i, k) ^ D and {k,j) ^ D for any i > k > j. By R{D) denote the set of all D-regular roots. 

Example 1.11. Let n = 6; D = {(3,2), (6,4)} C $(6) is a basic subset. Here we mark the 
roots (i, j) ^ R{D): 
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1 2 3 4 5 6 

1 

2 
3 
4 
5 
6 

Now, set ^reg = {(yhj) ^ ^{n) I i > n — j + 1}, rriD = \R{D) fl $reg| for an arbitrary regular 
subset D C Fix also any non-trivial character Fg ^ C of the additive group of the 

ground field. 

Theorem 1.12. Let Q = Qf C be a regular orbit, let / = be the canonical form 

of this orbit and x = X/ be the corresponding character. Then 

1. Supp X = Xvcg- 

2- x{g) = (1'"^° ■ (^fieniv)) for any g G Xoiv) C 3Creg, where ez)(v?) = Xd{'^) - In e Qn{q) 
and : 0„(g) ^ C is given by the formula 

ej{x) = o{f{x)) = Yl ^(Qi^ij), X = i^ij) e fln(g)- 



Proof. This is a special case of |A2l Theorem 5.1]. □ 

In Andre's work |A2| one can find explicit description of Xd^lp) for any basic subset D C 
$(n). More precisely, g G %d{'^) C Gn if and only if 

/^RD{^,3)^g) = A^^(*'^)(x^(v9)) (2) 

for all G R{D). Here RoiiJ) = U {{k,l) e D \ I > j and k < i}. 

In particular, if m = max(jj)g£) j then 

i/ij = if j>m ov i<n — m + 1. (3) 

One can see that characters of the principal series and regular orbits can be described in terms 
of minors of matrices from Gn, by definition, this is also true for all Andre's basic characters. 
In the next section we'll see that this is not true for subregular characters in general (see |A2| ). 

2. Subregular characters (statements) 

It follows from Theorem ll.4l that a coadjoint orbit "in general position" has maximum dimension 
2/i(n). More precisely, the set of points such that their orbits are regular is a dense open subset 
of Qni^l) Zariski topology. Denote this subset by Creg, it's given by the set of inequalities 

7^ 0, 1 ^ d ^ ni (thats's why we use the term "regular orbits"). Recall that n = UQ + rii + l, 
where Uq = [n/2], rii = [{n — l)/2]. 

On the other hand, for an arbitrary 1 ^ d ^ rii one can consider the hyper-surface Od C 3* 
given by = 0. This hyper-surface splits into the union of certain orbits. 

Definition 2.1. An orbit G g* (and the corresponding representation Tq and character 
Xn) is called subregular, if dimfi = 2/i(n) — 2 (resp., dimT^ = g'^^")"^), i.e., it has dimension = 
dimension of a regular orbit — 2 (in the other words, this orbit has sub-maximum dimension, 
because any orbit is even-dimensional |K4| ). 
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Each subregular orbits contains in the unique Od and has maximum dimension among all 
orbits containing in Od- Hence, we say that a subregular orbit is d-suhregular if it contains in 
Vt C Od- In [IP] defining equations of subregular orbits are found, but we will not use them in 
the sequel; we only list here elements of 0* such that their orbits are subregular. 

Definition 2.2. An element / = {^ij) G 0* is called a canonical form of a subregular orbit 
(of the first, second or third type resp.) if 

1. There are a number 1 ^ d < rii and . . . , Pd-i, P', P" , Pd+i, - - -, Pno-i ^ k*, Pno,P ^ ^) 
where Pno for odd n, such that ^j^n-j+i = Pj for any 1 ^ j ^ d — 1 and d + 2 ^ j ^ no, 

id,n-d = P' -, id+l,n-d+\ = P" ^ ^n- 



-j+i = Pj for any 1 ^ j ^ d- 
-d,n-d+i = P, and = for all other roots. 



2. n is odd and there are Pi,---,Pni-i € k*,P',P" G 
1 ^ j ^ ni - 1, ^nuno+i = P\^m+i,no+2 = P" ,and ^ij -- 



k such that C,j,n-j+i = 
= for all other roots. 



Pj for any 



3. n is even and there are Pi, - - - , Pm-i, P G k*,P',P" G k, such that ^j^n-j+i = Pj for any 
1 ^ j ^ ni - 1 and either ^m,no+i = P,^ni+i,no+2 = P' , ^ni+2,no+2 = P" (and all other 
iij = 0), or in^^no = P', ^ni+i,no+2 = P (and all other = 0). 

Example 2.3. Here we draw some canonical forms of subregular orbits for n = 8, d = 1 
and d = 2. The symbol ® marks the roots such that 7^ 0: 

1 2 3 45678 12345678 
1 ' ' ^ 
2 
3 
4 
5 
6 
7 
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Theorem 2.4. The orbit of a canonical form is subregular. More over, for each subregular 
orbit there exists the unique canonical form that contains in this orbit. 

Proof. See [IP] (there the case char A; = is considered, but the proof is still valid in our 
case A; = Fq if char k = n). □ 

This allows to define (i-subregular forms and subregular orbits of the first, second or third 
type by the obvious way. Clearly, if 1 ^ d < rii, then (i-subregular orbit is of the first type, and 
ni-subregular orbits are of the second type, if n is odd, and the third one, if n is even. 

So, the main goal is to find exact formulas for all characters of the form x/, where / is a 
canonical form of a subregular orbit. As for characters of the principal series, we'll describe 
the support of a given subregular character and compute the value of this characters on an 
arbitrary conjugacy class contained in the support (we'll firstly consider the case 1 ^ d < rii 
and then (in section [6]) the case of ni-subregular orbits). 

Notation 2.5. For an arbitrary 1 ^ ci < ni by Do^d) and Di{d) denote one of the following 
sets resp.: 

0, {{n ~ d + 1, d)}, {{n - d, d)}, {{n - d + l,d + 1)}, 
{{n — d,d), {n — d + l,d + 1)}, 
{{d+ 1, d) , {n — d + 1, n — d)} , 

{{d + l,d),{n — d + l,n — d),{n — d,d),{n — d + l,d + 1)}. 



Do{d) 
Di{d) 
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Definition 2.6. A subset D C $(?t,) is called d-subregular (1 ^ d < ni) if it has the form 
D = D' U Di{d), where either i = or i = 1 and D' is a regular subset that doesn't consist the 
roots {n — d + l,d) and {n — d + l,d + 1). Note that subregular subsets consisting Di{d) are 
not basic in general. 

Remark 2.7. Denote Supp/ = G $(n)) | /(ejj) 7^ 0}. One can see that D C 

Supp / U(n — (i + 1, n — (i) U (n — + 1, (i), if / is a canonical form of a subregular orbit. 
Example 2.8. Here we draw one of 1-subregular subsets for n = 6: 
D = {(2, 1), (5, 1), (6, 2), (4, 3), (6, 5)} C $(6). 
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Define R{D), xoi'^), eD(v'), 3Cd(v^), ^ ^iid ^Z? ^is for regular orbits. For a subregular orbit 
flf C Od denote 

where the union is over all (i-subregular subsets D and over maps ip: D ^ F* such that 

id,n-d ■ ^{d + 1, d) = ^d+i,n-d+i ■ '■p(n -d+l,n- d), where / = {^ij). (4) 
For a (i-subregular subset D containing Di{d) we denote 

'^d = e \ i> n- ] + 1,] ^ {d,n-d},i i {n - d + l,n - d}}. 

Finally, for a d-subregular subset D let 

f|i?(Z})n$,eg| -1, \{D^Do{d), 

mn = < 

\\R{D) n <^d\ + n - 2d - 1, ifDDDi(rf). 

Theorem 2.9. Let 1 ^ (i < ni, let Qf C Od C 0^(9) be a subregular orbit, and x = X/ be 
the corresponding character. Then 

1. SuppX = Xf. 

2. = g""^ ■ df{eD{^)) for an arbitrary g G Xd(v2) C X/. 

Sketch of the proof. In Lemma 13.21 we present certain equations the element xd{^) 
satisfies, and show that the ideal J of k[Gri\ generated by these equations is invariant under 
the adjoint action. In Lemma [3.31 we prove that J is a prime ideal. In Lemma [3.41 we find the 
stabilizer C = StabG„(a;_D(<y^)) and prove that 

dimXaif) = codimC = dimV^(J) 

(here V{J) = {g e Gn \ F{g) = for all F G J}, as usual). This shows that Xoi^p) = V{J), 
i.e., our equations are exactly the defining equations for the conjugacy class of the element 
xd{(p) (Theorem 13.51) . 



6 



On the other hand, in Lemmas 15.11 — 15.31 we prove that if xid) 0) then g satisfies these 
equations (and the additional equation ([4])), and compute the value xid) iii this case. This 
concludes the proof. □ 

Remark 2.10. li D D Di{d) is (i-subregular, ip are maps from D to F*, and 

ip{d +l,d) ■ ipin — d + l^d + 1) + ip{n — d,d) ■ (p{n — d + l,n — d) = 
= ip{d + l,d) ■ (p{n — d + l,d + 1) + ip{n — d,d) ■ (p{n — d + l,n — d), 

then = OCoiv) (see section [3]). But one can see (using the results of section [S]) that 

in this case 9f{eD{ip)) = Of^eoisp))- Since m^) is independent of y?, the character value in the 
formulation of the Theorem is well-defined. 

It's easy to prove a variant of this Theorem for the case of rii-subregular orbits by some 
modifications of definitions and formulations (see section Ej). 

3. Conjugacy classes 

Here we present an explicit description of the conjugacy class of an element xd{^)- Fix 

an arbitrary 1 ^ d < rii and a subregular subset D D Di{d) (in this section and in the two next 
sections we consider the case D{d) = Di{d); the case D{d) = Doi^d) is similar, see section [6]). 
Notation 3.1. It's convenient to split a (i-subregular subset into the union 

D = D-UDi{d)UD+, 

where D~ = {{i,j) e D \ j < d}, = eD\d<j<n-d}. 

Recall that D' = D \ Di{d) = U is a regular subset (see Definition 12.61) . Let 
D" = D\ {{n - d,d),{n- d+l,d + 1)}) (it's a basic subset of <l>(n), and D' C D" C D). 

It's impossible to describe JCoi'^) in terms of minors of matrices g = {yij) G G„. Let 
G and m = max(j j)g£)+ j. Consider the following polynomials: 

n—d m n—d 

o^ij = X] yn-d+i,iyi,j, Pij = 5Z yiM,d^ 7 = yn-d+i,iyi,d 

l=n-m+l l=d+l l=d+l 

(more conceptual description of these polynomials is given in section [6l). For simplicity, we'll 
write instead of A^d"(*'^)(5(), and yfj.ya instead of yd+i,d, yn-d+i,n-d resp. {RD"{i,j) are 
defined similar to Roii,]) in ([2])). 

Let us now start with proving Theorem 12. 9[ Let Cq G fc, Ca,Cp,Cij G fc*, where (i,j) G D' , 
are arbitrary scalars. Consider the ideal J of /c[G'„] generated by the elements 

ya - Ca,yf3 -C[^,-f- Cq, 

Aij-dj, {i,j)eD', 

it,j)eRiD")\D'. 

Lemma 3.2. J is Gn-invariant, i.e., if (? G V{J), then xgx'^ G V{J) for all x G Gn- 
Proof. Since the set D" is basic, all Aj^ invariant |A2l Lemma 2.1]. 
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Let g = {uij) G V{J). Denote Xrs{\) = 1 + Xcrs- Since every element x G Gn can be written 
as a product x = ^^^^^(Ai) . . . Xr^smi^m) for a certain m and (rj, Si) G $(^), Xi G k, it's enough 
to prove that if a; = Xrs{X), then xgx~^ G ^(J). But if x has this form, then 



{xgx 



'i/ij, if 2 7^ r and j 7^ s, 

if z = r and j ^ s, or j = s and i ^ r, 
yrj + XVsj, iii = r,j<s, 
^Uis-Xyir, if j = s,i>r. 



Hence, i/a, Up are invariant obviously, and the proof of invariance of other elements is by direct 
enumeration of possible values of r and s. 

For example, consider the polynomial Uij. If s ^ j or r > — d + 1, then all coordinate 
functions involved in this polynomial are invariant themselves; this is also true, ifr = n — d + 
s^n — m + l.lir = n — d+l,s>n — m + 1, then 



s—l n—d 



aij{xgx^)= ^ {yn-d+i,i + Xys,i)yi,j + ^yn-d+i,iyi,j 

l=n—m+l l=s 

s-1 

+ X] ys,iyi,j = aij{g), 



l=n—m+l 



because y^,/ = Oiora\\l^n — m + l^n — nQ + l>m (it's a particular case of ([3])). 

Ifr < n—d+1, s = j, then a^j is invariant for the same reasons. Finally, if r < n—d+1, s > j, 
then 



{xgx ^) = ^ yn-d+l,iyij + iyr,j + Xysj)yn-d+l,r + y s,j{yn-d+l,s - Xyn-d+l,r) = 

= aij{g) + X ■ {ysjyn-d+i,r - ys,jyn~d+i,r) = Olijig)- 

Invariance of (3ij and 7 can be proved similarly. □ 

This means that V{J) is a union of cojugacy classes. It's easy to see that xd{'^) G V{J)^ if 

Cij = Aij{xD{<^)), 

Co=-f{xD{v)), .gx 

Ca = (p{n — d + l,n — d), 
cp = (p{d + 1, d). 

Thus, Xd{<p) C V{J) for this values of constants. 

For any root ^ = (z, j) G we define its level as the number u{C,) = i — j. Then the 

formula 

^ = ihji) <V = (^2, J2) ^ either u(0 < u{ri) or u{^) = u{r]),ji < jz, (7) 

defines a complete order on the set of all roots. For each root ^ = G $(n) let be the 
ideal in k[Gn] generated by all y^^,!] < C,, and ^0 = (^O)Jo) be the maximal root in D", which is 
less than C, (if exists). 
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Lemma 3.3. The ideal J is a prime ideal of k[yij\ (here k is the algebraic closure of k). 
Proof. Consider the following transformation of coordinates: 

yn-d+l,n-d = Va — Cai yd+l,d = VfB — Cf3, Vn-d+lA+l = 7 ~ Cq, 

yn-d+i,i = aij, yi,d+i = Pij^ ihj) e D^, 
yij = - Cij, {hi) e D\ 

yi,=A,^, {i,3)eR{D")\D'. 

(it's easy to see that J = {y^)^^B, where B C $(n) denotes the set of all roots from the 
left-side hand of (El)). Note that for any ^ G i? we have 

y^ = yry^ + ^i (mod/^), (9) 

where is an invertible element of k[yij]/J, and G A; is a certain scalar. Indeed, this is 
evident for yn-d+i,n~d, yd+i,d- For other roots we have: 

yn-d+14+1 = 1 — Co = yn~d+l,d+iyd+l,d + • • • , 

yn-d+i,i = oiij = yn~d+i,iyi,j + • • • , (^, j) e 
yi,d+i = Pij = yi,d+iyd+i,d + • • • , (ij) & d^, 

yij ^ij Cjj yij ' ^ioj'o {h j) ^ ^ j 

yi, = = yi, ■ Ai„ j„ + . . . , e R{D") \ D', 

where for any y^, ^ & dots denote elements, equal to zero modulo and scalars (we assume 
that = 1, if the root does not exist for a given G $(72)). But one can easily obtain 

the following equatities modulo J: 

yd+l,d — Cf} 7^ 0, yn—m+l,m — Cn~m+l,m 7^ 0; 

yi,j = Cij/cio,jo 7^ 0, G J < m, 

^ioJo = Ciojo ^ 0, (io, jo) e D" 

(recall that m = max(j j). This concludes the proof of ([9]). 

Hence, in k[yij]/J all y^ are polynomial in y^ (for each ^ G 5). Consequently, 

k[y^j]/J= k[y^hG^(n)/ {m)(eB = ^[{j/J^^B U {y^j^es] /{m)^eB = Hyihts- 

In particular, k[yij]/J is a domain, so J is a prime ideal. □ 

This Lemma shows that V{J) is an irreducible subvariety in Gnik), because k[Gn] = k[yij]. 

Lemma 3.4. Let C = StabG„{x£){f)) = {g E Gn \ gxoif) = X£){f)g} be the stabilizer 
(the centralizer) of Xoif)- Then dimC = codim\/(J), where J is generated by the elements ((51) 
with the scalars given by ([6]). 

Proof. Firstly, we'll present the defining equations of the centralizer C. For any ^ = (z, j) G 
$(?7,), let $^ be the union of all roots form the i-th column and the j-th row. We put 

= {(n-rf + l,i) I (z,j) GD+}, = {(n-rf+ 1,j) \ j < d}, 
$^3 = {{i,d+l) I G D+}, $^ = {{i,d) \ t>n-d+l}, 
^^ = {{n-d, d)}, = (U^eD^c) n (RiD) \ D), 
A = <l>c, U U <l>^ U $^ U <l>^ U $5 C $(n) 
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(these sets are really disjoint). It's easy to check that C is given by the following equations: 

(a) yn~d+i,iaij = an-d+i,n-dyn-d,j, {iJ)eD^, 
iP) yi,d+iO'd+i,d = (^ijUjd, 

(a) an-d+l,n-dyn-d,j + (ln-d+l,d+iyd+l,j =0, j < d, ^^^^ 

{(3) yi^d+iad+i,d + yi,n-dan-d,d = ^i i > n - d+1, 

(7) yn-d+l,n-dO'n-d,d + yn-d+l,d+lO'd+l,d = 0'n-d+l,n-dyn-d,d + (^n-d+l^d+lVd+l^-, 

{5) yij = 0, {i,j)e<^s, 

(here we write aij = G k* for simplicity). 

So, C is defined by the equations (fTOl) . which are labeled by roots from A, and V{J) is defined 
by the equations ([5|), which are labeled by roots from B. Consider the maps aA- A $(n) 
and as - B ^ ^{n), given by the formulas 



MO 



{n-i + l,j), 
ihn-j + 1), 
(z,d), 

{n -d+l,j), 
{n — d + l,d), 



^ = {t,j),{n-i + l,t)eD",j y^d+l,t<n~j + l, 

^ = ihj), + 1, j) e D",i ^n - d,i> n- i + 1, 

^ = (i^d+l) e ^s,i ^ n~ d, 

^ = {n-d,j)e<^s,Jl^d + l, 

^ = (n - c/,rf + 1) G $5, 

elsewhere, 



^^(0 = I ^ = {n-d + l,i), G D+, 

1 ^, elsewhere. 

They define the map a: AU B ^ $(n) (here A U 5 is the disjoint union of A and B) by the 
rule (j\a = ^A^ ~ ^B] it's easy to see that this map is a bijection. Since dim3C£)((/9) = 
codimC =1^1, codimV^(J) = |-B|, this concludes the proof. □ 

Theorem 3.5. The defining ideal J of the conjugacy call %£,{ip) C Gn{k) of an element 
XD^ip) is generated by the elements (JH]) with the scalars given by ([6]). 

Proof. Since conjugacy classes of G„(fc) are Zariski-closed [Sl Proposition 2. 5], previous 
Lemmas follow that ^^(v^) = V{J) over k. Hence, the sets of their fc-points also coincide. □ 

4. A semi-direct decomposition of Gn 

The next goal is to describe the support of a subregular character x = Xf explicitly (we'll see 
that the support coincides with %f ) and to compute the value of this character on a counjugacy 
class contained in the support. We'll use induction on dimension of the group and the Mackey's 
method of semi-direct decomposition (see, f.e., |Le|). In this section we collect some basic facts 
which are needed for the sequel. 

Let G be a finite group, A, B be its subgroups and G = A >i B he their semi-direct product 
(i.e., G = AB and A<G). 

Definition 4.1. Let G = A >i B he a finite group and A he abelian. For a given irreducible 
character ip of the group A, the subset B"^ = {b E B \ ip o = ip} oi the group B is said to be 
the centralizer of this character (here r^: A ^ A: a bab^^). 
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The following Theorem satisfies |Le| : 



Theorem 4.2. Let G = A yi B he a finite group and A be abelian. Then every irreducible 
representation r of the group G has the form r = IndJ^^^xV" ® 'T^j where ^/^ is a certain irreducible 
character of the group A, and r is a certain irreducible representation of the centralizer . 
Hence, every irreducible character x of the group G has the form x = ^^'^axb^I'^X, where x, tjj 
are certain irreducible characters of the groups A, B^ resp. On the other hand, any character 
Ind^^^v^X is an irreducible character of G. □ 

We denote 

Pn = {9 = iVij) G Gn I Vij = for j ^ 1}, 

Gn-i = {9 = iVij) e Gn I Vij = for j = 1} ^ Gn- 

Then Gn = Pn ^ Gn-i, moreover, the group P„ = F^"^ is abelian; hence, all conditions of 
Theorem 14.21 are satisfied. 

We fix a non-trivial additive character 9: ¥q — > C. Any irreducible character of the group 
Pn has the form 

P = (Pij) e Pn^ 0{s2V2l) ■ ■■■■ diSnPnl), 

where Sj G F^, 2 ^ i ^ n, are arbitrary scalars. We'll consider the case 9n{p) = 9{snPni) and 
the case 9n-i{p) = 9{sn-iPn-i,i)- One can see that their centralizers in the subgroup Gn-i are 

G^"-! = {9= iVij) e Gn^l C Gn I Vnj = for 1 ^ J ^ 72 - 1}, 

C'r/ = {9 = {y^j) e Gn-i C Gn \ Vn-ij = for 1 ^ J ^ n - 2}. 

Note that G^'li = Gn-2, and G^Ti^ = Gn-2 = Gn~.2 x IFg (we'll consider only these embeddings 
of these subgroups into G„). 

For a linear function / G g*, let 7r(/) G fl*_2 and vf(/) denote its restrictions to gn-2 and 
5„_2 respectively (here 0„_2 = Lie G'ri-2 and embeddings of subalgebras into g„ correspond to 
embeddings of subgroups into Gn)- Due to the Mackey's method, if 1 ^ d < ni and / = {^ij) G 
Od is the canonical form of a ci-subregular orbit, then 

Tf = Indp^^f.^ ,,6'„ O T^(/), where s„ = if d > 1, 

T/ = Indp" ~ 9n-i ® T5f(/), where = ^i,„_i, if = 1, 

where (resp. TT^{f) and T^{f)) denotes the representation of the group Gn (resp. G„_2 and 
G'„_2), corresponding to the orbit i7/ C g* (resp. ^^7r(/) C g,*_2 and f^?F(/) C g5^_2)- terms 
in the last equality are well-defined, because the orbit method is valid for the group Gn-2- So, 
the problem can be reduced to the study of representations of dimension less than the first one 
has, and we can use an inductive argument. 

Finally, certain coset decompositions of the group G„ are needed for construction of induced 
representations. It's easy to check that 

Hn = {h = {tij) G Gn-i I tij = for z 7^ n], 
Hn = {h = (tij) G Gn-i I tij = for i 7^ n - 1}. 

are complete systems of representatives of Gn/{Pn x Gn~2) and Gn/{Pn x Gn-2) resp. 

Remark 4.3. Since Gn-2 — Gn-2 xlFg, we have a complete description of the representation 
T^(/). Precisely, we note that Gn-2 is isomorphic to the subgroup Gn-2 x of G„_2 (this induces 
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the embedding of Lie algebras Qn-2 ^ 0n-2); let ip be the projection Gn-2 Gn-2 = Gn-2 x 
and g = {yij) G Gn-2- Then the character of the representation T^(/) has the form g i— ^ 
x(^((7)) ■ 0{^n-i,nyn,n-i), whcrc X IS the character of the principal series of Gn-2, corresponding 
to the orbit 

Remark 4.4. There is another ("symmetric") decomposition G„ = x G^_]^, where 

K = {9 = iVij) e Gn I Vtj = for J ^ 1}, 
G'n-i = {9 = iVij) e Gn I Vij = for j = 1}. 

Reflecting other subgroups and subsets in the anti-diagonal, we get G^_2, H'^^ and H'^ {Gn-2, 
embedded into G„ as above, is invariant under this reflection, i.e, G^_2 = Gn-2)- Then the 
irreducible character 9'n_i : P^^C has the form Pn 3 p = (pij) 0{^2,nPn,2 + ^n-i,nPn,n-i) ■ 
Now we are able to conclude the proof of Theorem I2.9[ 

5. The proof of the Main Theorem 

Let 1 ^ d < rii, f E be a canonical form of a c/-subregular orbit and X = X/ be the 
corresponding character (as above). We'll prove in this section that its support coincides with 
%f and compute its value on an arbitrary conjugacy class Xd{^) C %f. We've proved in 
section [3] that the subvariety %D{ip) of Gn is defined by the equations (JS]) with the scalars given 
by ([6]); in the other words, V{J) is the defining ideal of %£i{ip) in k[Gn]- 

The following proof is by induction on n; the base can be checked directly (f.e., using 
We'll assume that if g = (yij) G G„, then yn-d+i,n-d 7^ (this means that D D Di{d)). Firstly, 
we have to consider the case d = 1. For convenience, we denote 

= {(^.J) e $(n) I i = zo}, $io = Ui^io^i. 

Lemma 5.1. Let / G Oi be a canonical form of a 1-subregular orbit, and g G G„. If 
x{g) 7^ 0, then g G 3C£)((y9) for a certain Xoif) satisfying (Jll). 
Proof. Recall that 

(see (fTTl) ). Since Gn = Pn^ G„_i, an arbitrary element g E Gn can be uniquely represented as 
g = pg', p G Pn, g' G Gn-i (in fact, p and (7' are given by replacing the corresponding entries 
of g by zeroes). Hence, for a given g G G„, the element p{g) G Pn is well-defined. By T^{g) 
we denote the element of Gn-2, which is given by replacing the corresponding entries of g by 
zeroes. We have 

X{9) = Xfi9) = Ind^" G .^n-M9)) ■ X^(/)(^(^)) = Yl ^n-Mh-^9h)) ■ Xnid^ih-^9h)) 

(12) 

(here the summation is over all h G Hn such that h ^gh E Pn >i Gn-2)- 
Note that for any h = (tij) G Hn, 

{h-^gh)nj = VnJ + yn,n-ltn-l,j, if 2 ^ J ^ 72 - 2, 

n-2 

ih~^gh)n-i,j = Vn-ij - ^ tn-i,iyi,i, if 1 ^ j ^ n - 3, (13) 

i=j+l 

{h~^gh)ij = yij for all other 1 ^ j < i ^ n. 
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If Xn{f){7^{g)) 7^ 0, then it follows from the equations of regular orbits (Theorem 11.41) . formulas 
for characters of the principal series (Theorem I1.12p and remark 14.31 that 

x^ni^g)) = q' ■ x.HniAg)) ■ ^(6,n ■ (-i)i^^'-^ • ^3^) ■ e{^r.-i,nyn,n-i). (14) 

Here Xi = D \ ($i U $"-^), X2 = Xi \ s = \{R{D+) n $„) \ ($2 u $"-i)| and tt^ denotes 
the maps G„ Gn-4 and g* 0*_4 (we assume that Gn-4 is embedded into G„ like Gn-4 C 
Gn-2 C Gn). In particular, Xn'^if) is a character of the principal series of the group G'n-4- 

On the other hand, [13]) shows that Xn'2(f){'^'^{g)) = XTT^if){'^'^{h^^gh)) for all h G Hn- 
Substituting this to f fT2i l and using ([T4ll . we obtain 



5^ ^„.l(j>(/.-l^/^)) . ^(6,„ ■ (-1)1^^1-^ ■ ^^S^, + ^n-l,nyn,n-l). ^^^^ 



Furthermore, 

n-2 

9n-l{p{h~^gh)) = 9{^l,n-l ■ (l/n-1,1 - ^ ^n-l,il/i,l)), 

*=2 (16) 

A^^ jh-'gh) A^^ jg) + ■ A^^ (^,) 

A^^{h-^gh) A^2(^) 

where denotes the matrix given by replacing Unj by tnj for 2 ^ j ^ n — 2 in the matrix g 
(the last equality can be checked directly). 

Suppose now that x{.g) 7^ 0- Thus x-n'^{f){.'^'^{.g)) 7^ (see (fTSll ). some conditions from the 
equations ((51), defining 3Ci:)(v3), are satisfied automatically. Precisely, we have to check only the 
following equalities: 



^l,n-l2/2,l = 6,nyn,n-l, (couditiou ^) 



a 



A, =0, {i,])eD' 



Ai, = 0, (z, j) G (/2(D") \ D') n ($2 u $„_i). 

But (ra,n - 1), (2, 1) G D", so i?(D") H ($^ U $„) = 0, hence, the last set of equalities in fact 
has the form A„_i = 0, Aj 2 = 0. 

Consider the equalities A„_i j = firstly. If j > m = max(j j)^^) j, then they have the form 
A„_i j = yn-i,j = 0, and other equalities (with j ^ m) are exactly Kronecker-Capelli conditions 
of compatibility of the system of linear in tn-ij equations 

n-2 

yn-i,j - ^ tn-i,iyi,i = 0, 2^j^m. (17) 
i=j+i 

In both cases equalities under consideration are satisfied, because they express the condition 

h-^gh G P„ X G„_2 (see pD). 



On the other hand, it follows from this condition and ( fT3ll that t„_i , = — ^^^^ for all 
m < j ^ n — 2. Substituting this to ( flTl) . we obtain that aij = for any (z, j) G (all other 
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equalities of this form are satisfied because of compatibility of the system (flTll). The equalities 
Pij = 0, (z,j) G D~^, and the other equalities A, 2 = can be obtained by using symmetric 
semi-direct decomposition of G„ (see remark 1441) . 

Finally, we'll prove that 1^ is satisfied. Note that ^^gf ^i^) = ^ ^ Hence, 

the coefficients of j in the character formula, that vary independently, have to be zero. For 
example, the coefficient of tn-1,2 equals zero. Since A^^{g^) = (— . 2 ■ A-^^{g) + . . . 
(members without tn-1,2), the coefficient of t„_i,2 in the character formula is equal to 

~^l,n-iy21 + ^2,n ' ( — 1)'"^^' " l/n,n-l ' A^^^J^ ~ ~^l,n-iy21 + ^2,nyn,n-l, 

SO, -6,„_iy21 + ^2,nyn,n~l = 0. □ 

Lemma 5.2. Let / G Ci be a canonical form of a 1-subregular orbit, g G %d{(p) C Xf. 

Then x{g) = q""' -OfieDi^)). 

Proof. We'll compute the value of the character under the assumption that this value isn't 
equal to zero. Substituting (fT6l ) to ( |T5l) . we have: 

n-2 

Xig) = ■ X^2(/)(7r^(^)) ■ 0{^l,n-l ■ {yn-1,1 - ^tn-l,iZ/i,l) + 

1=2 



heH„ 

, . ( tV^^I ^'''i9)+yn,n-i-A^^{g,) , 

+ <;2,n ■ I — -Lj ■ A^^2"(^) ^n-l,nyn,n-lj 



if X5f(j)(7f ((yf)) 7^ 0. If x{9) 7^ 0, then the following four groups of conditions are satisfied. 

1. The first condition x-K'^(f){.'^'^{.9)) 7^ is satisfied automatically, because g G %D{ip). 
Indeed, it's evident, that vr^(Z}) G $(n — 4) is a regular subset and '^'^{g) G X7r2(z))(v2 \-k'2(d))- 
But X7r2(/)(7r^(5')) is a character of the principal series of G„_4, so. Theorem 11.121 shows that 

X.^(/)(^'(^7)) = r-^<-'-^.^(/)(7r^(eD(0))) 

(the map tt^: g„ — »• Qn-i is defined by the obvious way). 

2. The second group of conditions provides the inequality Xn(j)(!^{.9)) 0- Since x^2(j)(7r^(5f)) 
7^ 0, it's enough to prove that Aij{h~^gh) = for G {R{D")\D)r\{^'^U^n) ■ More precisely, 
these conditions have the form 

A,,2(^7) = 0, {i,2)eR{D)"\D 

Anj{h~^gh) = Anj{g) + yn,n-i ■ Anjigt) = 0, (n, j) G R{D)" \ D. 

The first set of equalilies is contained in the Kronecker-Capelli conditions of compatibility of 
the system (fT71) . Since tn-ij, 3 ^ j ^ m, contain only in the second set of equalities, it follows 
that tn-i,j, 3 ^ j ^ m, (n — 1, j) ^ R{D"), may be arbitrary, and other tn-ij are uniquely 
determined by the formulas 

-AnM - E r>J ±tn-l,r " A^^'^^^g) 

+ I i\|Xi|-3 {n-\,r)iR{D") , . 

'"--'^^ - ^-'^ yn,n-, ■ An.) ' (18) 

where F(j) = Fr(j) = (-D+\($''U$-'^^))U{(n-r + l, j)}, and signs of alternates. 

3. The third condition says that we must consider h G such that h^^^gh G ^ Gn-2- 
This condition is always satisfied, if g E Xni'^) (see the proof of Lemma [5T]) . 
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4. Finally, the fourth condition says that the coefficients of tn-ij in the character formula, 
that vary independently, are equal to zero. These are tn-1,2 and tn-i,j,S ^ j ^ m, (n — ^ 
R{D"). The end of the proof of Lemma ISTTl shows that the condition ([4]) is satisfied if and 
only if the coefficient of tn-1,2 equals zero; hence, we must prove only that the coefficients of 
all t„_ij,3 ^ j ^ m, (n — l,j) ^ R{D")^ are equal to zero. Simplifying (fTSl) under all our 
assumptions, one can check that this is equivalent to the set of equalities (3ij = 0, E , 
which are satisfied, because g G 

So, the character formula contains only the following expression: 



n-2 



yi,i ■ ( — ^)) + 6,n- 



I 1 yn,n—l 



En— i 
i=n-m+l yn,iyi,l Z^i=m+1 yn,iyi,l 

C,n-l,nyn,n-l + Sl,n-1 " r C,l,n-1 ' r ^2,n 

Z/n,7i— 1 2/n,n— 1 



(19) 



where the numerator of every summand of the group marked by dots contains the only one 
of the elements yi^i, 2 ^ i ^ n — m, and these elements aren't contained in other summands. 
On the other hand, the symmetric decomposition of the group Gn (see remark 14.41) give the 
following expression in the character formula 

En—m 

^n.-l,ral/n.,n-l + ^2,ra " r C,l,n-1 ■ ■ ■ ■ — 

^21 C2o^ 

_ j=2 ynjyjA ^ i^j=m+i ynjyj,! 

— ^n-l,nyn.n-l + «,n " r q,2,n ' r U,n-1 " • • • ; 

Z/21 1/21 

which has to coincide with (fT9l) (here the numerator of every summand of the group marked 
by dots contains the only one of the elements ra — m + l^j^n — 1, and these elements 
aren't contained in other summands). Since ^i,n-iy2i = C,2,nyn,n-i, we have that, in fact, (fTOl l 
and (I20I) have the common form 



En— 1 ^r^n—m / \ 

i=n-m+iyn,iyi,l . c 2^j=2 ynjyj,! ^ _ ^ 

Sl,n— 1 ' ~r S2,n ' ~r Sn— l,ni/n,n— 1 Sl,n— 1 ' ' Sn— l,ni/n,n— 1- 

yn,n-l 2/21 2/n,n-l 

We also have 

■ + en-l,n2/n,n-l) = " ^(6,n-l " + ^n-l.nZ/n.n-l) , (21) 

~ 2/n,n— 1 yn,n—l 

heHn 

where Si = |-R(i5)n$„_in$'"|. Indeed, tn-ij, 2 ^ j ^ m, (n — 1, j) G -R(-D), vary independently, 
and if we fix them, then all other tn-i,j are uniquely determined. 
Note that 

. 7(5') _^ 7(a;D (</?))_ ^ an-d+l,n-dan-d,l + an-d+l,2(l2,l _ 

?l,n-l ■ — Q,n-1 ■ — '51,n-l " — 

2/n,n— 1 2/n,n— 1 ^n,n— 1 

_ I ^l,n-l _ ^ G,n-d+l _ (22) 

— ^l,n-l'3.n-l,l H ■ an-d+1,20'2,1 — Q,n-l«.n-l,l H ■ 0^-^+1,202,1 — 

'3'n,n-l ^2,1 

= G,ra-lCtn-l,l + 'C2,n-d+ian-d+l,2 

(here atj = (p{i,j), as above). 
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Finally, substitute fl2Tll and (l22l) to ffTSD: 

Xnj{g) = q' ■ x^2(/)(7r^(^)) ■ ■ 



2/n,n— 1 



lid) 




because m^2(£)) + s + si = ttid- The proof is complete. □ 

Lemma 5.3. Let 1 < d < rii, f E Od he a canonical form of a (i-subregular, x = Xf he the 
corresponding character, and g E Gn- Then xid) 7^ if and only if g E Xnif) C Xf; in this 



(here the summation is over h E Hn such that h~^gh E Pn >^ Gn-2)- Since 7r(/) is a canonical 
form of a subregular orbit oiGn-2, we may assume that Theorem l2.9l is valid for X-nij) (inductive 
assumption). 

Using the same arguments, as in two previous Lemmas (Kronecker-Capelli criterion of 
compatibility of systems of linear equations, conditions of the form h'^gh E Pn ><i Gn-2, and 
vanishing the coefficients of independent variables in the character formula), one can obtain 
the required result. □ 



Above we considere c/-subregular subsets D with D{d) = Di{d) and assume that d < rii. Here 
we'll consider all other cases. 

First, if D D DQ^d), then the subset D is basic and the conjugacy class JCD^ip) is given by 
the equations ^ (see fA2]). 

Second, we should consider subregular orbits of the second and the third types (these are 
ni-subregular orbits for even and odd n respectively). Note that if / = {^ij) E is a canonical 
form of a subregular orbit of the second type, then the set D = E $(n) | ^ 0)} is 

basic itself, so in this case the subregular character is a basic character (in the sense of Andre); 
a complete description of such characters is given in {A2| . This is also true for subregular orbits 
of the third type, if ^ni,no+i = (see definition 12. 2p . 

If O/ is a subregular orbit of the third type and 7^ 0, then all statements are the 

same as in the case of orbits of the first type (except that the ideal J doesn't contain the 
polynomial 7 — Cq). All proofs are quite similar to proofs in case of orbits of the first type, so 
we don't reply them. 

Finally, we should explain the genesis of polynomials aij, Pij and 7 from our description of 
subregular characters of the first type with D D Di{d): (these polynomials are not minors!). 
Consider the following characteristic matrix 




X{g)= ^OMh ^gh))-Xnif){T^{h ^gh)) 



6. Remarks and generalizations 



/ 1 

t ■ y2i 





1 












M{t) 



t ■ Vn-IA t ■ yn-l,2 

\ t ■ yn,i t ■ yn.2 



t ■ 



1 

yn,n-l 1 / 
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A minor of the matrix M{t) is polynomial on t, and its coefficient are polynomial on t/ij 
(hence, we can consider them as function on Gn or as elements of Fg[gn]). 

Notation 6.1. By MjJ'''''*^(t'^, gf) we denote the value of the coefficient of in the minor 
of the matrix M{t) with rows 1 ^ ii, . . . ,ik ^ n and columns 1 ^ ji, . . . , jk ^ n on an element 
g = (yij) G Gn- We also denote M^{d,g) = M^^j^^'"'''^'^^{t'^, g), where X is the set of pairs of the 
form X = {(«!, ji), . . . , {ik,jk)}, and a, r are permutations such that <j{ii) < . . . < a{ik) and 
r(ji) < . . . < r(jfc). 

For example, for any 1 ^ d ^ no we have M^{d,g) = Ad{g), where X = {(n, 1), (n — 
1, 2), . . . , (ni + 2, no)}; hence, all minors considered above are also coefficients of minors of 
characteristic matrix. On the other hand, if G (where D D Di{d) is a li-subregular 
subset and 1 ^ d < rii), m = max(jj)gD j, then 

«,,((?) = ±M^5 (2,(7), xr^ = {n-d+l,j}U (j 
A,(^7) = ±m4(2,^?), 4 = {z,4U U {(j,j)}, 

d<j sCm 

j{g) = ±M''\2,g), X^ = {n - d + 1, d} U [j 

d<t<n— d+l 

(the choice of signs depends on the cardinality of D^). 

We see that subregular characters can be described in terms of coefficients of minors of the 
characteristic matrix. In fact, subregular orbits, all orbits for n ^ 7 [IP] and all irreducible 
characters for n ^ 5 [I] can be described in these terms too. So, we may conjecture that all 
orbits and characters of the unitriangular group of an arbitrary dimension can be describes in 
terms of coefficients of minors of the characteristic matrix. 

The author expresses his sincere gratitude to professor A.N. Panov for suggesting the ideas 
that gave rise to this work. 
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